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f^ ' Abstract Five ordering algorithms for the nonserial dynamic programming algo- 

Cn ' rithm for solving sparse discrete optimization problems are compared in this paper 

The benchmarking reveals that the ordering of the variables has a significant impact 
on the run-time of these algorithms. In addition, it is shown that different order- 
ings are most effective for different classes of problems. Finally, it is shown that, 
amongst the algorithms considered here, heuristics based on maximum cardinality 
jy^ ' search and minimum fill-in perform best for solving the discrete optimization prob- 

O . lems considered in this paper. 
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The use of discrete optimization (DO) models and algorithms makes it possible 
to solve many real-life problems in scheduling theory, optimization on networks, 
routing in communication networks, facility location in enterprize resource planing, 
and logistics. Applications of DO in the artificial intelligence field include theo- 
rem proving, SAT in propositional logic, robotics problems, inference calculation in 
Bayesian networks, scheduling, and others. 
- T— , , Many real-life discrete optimization problems (DOPs) contain a huge number of 
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variables and/or constraints that make the models intractable for currently available 
DO solvers. Usually, such problems have a special structure, and the matrices of 
constraints for large-scale problems are sparse. The nonzero elements of the matri- 
ces often involve a limited number of blocks. The block form of many DO problems 
is usually caused by the weak connectedness of subsystems of real-world systems. 
One of the promising ways to exploit sparsity for solving sparse DOPs is nonserial 
dynamic programming (NSDP) 121, ifTSl . NSDP eliminates variables of DOP using 
an elimination order which makes significant impact on running time. As finding an 
optimal ordering is NP-complete |fT4|, heuristics are utilized in practice for finding 
elimination orderings. The literature has reported extensive computational results 
for the use of different ordering heuristics in the solution of systems of equations 
im, Q. However, no such experiments have been reported for NSDP to date. 
Given the increased recent interest in DOPs, the subject of experimental research of 
NSDP algorithms that utilize heuristic variable orderings is timely. In this paper, we 
present comparative computational results from the benchmarking of five ordering 
techniques, namely: minimum degree ordering, nested dissection ordering, maxi- 
mum cardinality search, minimum fill-in, and lexicographic breadth-first search. 



2 Nonserial Dynamic Programming Algorithm 

Consider a DOP with constraints: 

keK 

subject to the constraints 

gi{Xs,)RiQ, /GM={l,2,...,m}, (2) 

Xj&Dj, ;G A' ={!,...,«}, (3) 

where 

X = {xi,. . . ,x„} is a set of discrete variables, F*' C {xi,a:2,... ,x„},A:GAr=:{l,2,...,f}, 
t - number of components of objective function. Si C {1,2,. ..,«}, R, 6 {<,=,> 
}, / G M; Dj is a finite set of admissible values of variable xy, j G A^. The functions 
/a(X*), A: G /T are called components of the objective function and can be defined 
in tabular form. We use here a notation: if 5 = {y'l,... Jq} thenX^ = {xj^,- ■ ■ ^Xj }. 



2.1 The structure of sparse discrete optimization problems 

Let us take a detailed look at an NSDP implementation for solving DO problems for 
the case when the structural graph is an interaction graph of variables. 
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Definition 1. ||2l. Variables x ^ X and y ^ X interact in DOP with constraints (we 
denote x ^y)if they appear both either in the same component of objective function, 
or in the same constraint (in other words, if variables are both either in a set X'^, or 
in a setX^.). 

Definition 2. Q. Interaction graph of the DOP is called an undirected graph G = 
{X,E), such that 

1. Vertices X of G correspond to variables of the DOP; 

2. Two vertices of G are adjacent if corresponding variables interact. 

Further, we will use the notion of vertices that correspond one-to-one to variables. 

Definition 3. Set of variables interacting with a variable x G X, is denoted as Nb{x) 
and called a neighborhood of the variable x. For corresponding vertices of G a neigh- 
borhood of a vertex x is a set of vertices of interaction graph that are linked by edges 
with X. Denote the latter neighborhood as Nb(}{x). 

In hypergraph representation of DO problems structure, the set of vertices H of 
hypergraph, equals to the set of variables X from the DO problems, and hypergraph's 
hyperedges forms subsets of related variables that are included in constraints, which 
means the hyperedge defines constraint scope. 



2.2 NSDP (Variable elimination) algorithms 

Consider a sparse discrete optimization problem ([T]i - © whose structure is de- 
scribed by an undirected interaction graph G = {X,E). Solve this problem with a 
NSDP 

Given ordering of variable indices a the NSDP proceeds in the following way: it 
subsequently eliminates Xa, , . . . ,Xa„ in the current graph and computes an associ- 
ated local information hi{Nb{xai)) about vertices from Nbixa^) (i~ 1, . . . ,«). This 
can be described by the combinatorial elimination process: 

G"=G,G^...,G^■-^G^...,G'^ 

where G^ is the x^ -elimination graph of G'^' and G" = 

The process on interaction graph transformation corresponding to the NSDP scheme 
is known as elimination game which was first introduced by Parter ifTOl as a graph 
analogy of Gaussian elimination. The input of the elimination game is a graph G 
and an ordering a of G. 

Consider the DOP described above and suppose without loss of generality that vari- 
ables are eliminated in the order xi , . . . ,x„ . Using the variable elimination scheme 
eliminate a first variable x\. This xi is in a set of constraints with the indices of 
U\ = {i I xi G Xsf}. Together with x\, in constraints Ui are variables fxom.Nb{xi). 
To the variable xj corresponds the following subproblem Pi: 
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hiiNbixi)) = max{ £ My'=) \ gi{Xs,) Ri 0, ieUu Xj e Dj, Xj G m[x^\y 



'' -kGK, 



Then the initial DOP can be transformed in the following way: 

max { £ my') I gi{Xs,) Rt 0, / £ M, xj e Dj, j e n} = 

max I r /i(}'*)+/ii(A?^(xi)|g,(^5,)/?,0, i eM-Ui, Xj eDj, j eN-{l}} 



keK-K, 



The last problem has « — 1 variables; from the initial DOP were excluded constraints 
with the indices from Ui and objective function term Y^keK, fk{Y'')', there appeared a 
new objective function term h\{Nb{x\)). Due to this fact the interaction graph asso- 
ciated with the new problem is changed: a vertex x\ is eliminated and its neighbors 
have become connected. 

Denote the new interaction graph G' and find all neighborhoods of variables in G' . 
NSDP eUminated the remaining variables one by one in an analogous manner 



3 Elimination Ordering Techniques 

An efficiency of the NSDP algorithm crucially depends on the interaction graph 
structure of a DOP. If the interaction graph is rather sparse or, in other words, has a 
relatively small induced width, then the complexity of the algorithm is reasonable. 
At the same time an interaction graph leads us to another critical factor such as an 
elimination order which should be obtained from the interaction graph. 
From the other side the NSDP algorithm heavily depends on the elimination order- 
ing. A good elimination ordering yields small cliques during variable elimination. 
There are several successful schemes for finding a good ordering which we will used 
in this paper: minimum degree ordering algorithm (MD), nested dissection order- 
ing algorithm (ND), maximum cardinality search algorithm (MCS), minimum fill-in 
heuristic (MIN-FILL) and lexicographic breath-first search algorithm (LEX-BFS). 



3.1 Minimum degree ordering algorithm 

The minimum degree (MD) ordering algorithm IT] is one of the most widely used 
in Unear algebra heuristic, since it produces factors with relatively low fill-in on a 
wide range of matrices. 

In the minimum degree heuristic, a vertex v of minimum degree is chosen. The graph 
G', obtained by making the neighborhood of v a clique and then removing v and its 
incident edges, is built. Recursively, a chordal supergraph H' of G' is made with the 
heuristic. Then a chordal supergraph // of G is obtained, by adding v and its incident 
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edges from G to H'. To create an elimination order with help of minimum degree 
ordering algorithm the minimum jlegreejorderingi) function from BOOST library 
||3l has been used. 



3.2 Nested dissection algorithm 

To create an elimination order, we recursively partition the elimination graph us- 
ing nested dissection. More specifically, we use METIS-EdgeND{) function from 
METIS Ubrary Q to find a nested dissection ordering. 



3.3 Maximum cardinality search algorithm 

The Maximum Cardinality Search (MCS) algorithm |fT3l visits the vertices of a 
graph in an order such that at any point, a vertex is visited that has the largest number 
of visited neighbors. An MCS-ordering of a graph is an ordering of the vertices that 
can be generated by the Maximum Cardinality Search algorithm. The visited degree 
of a vertex v in an MCS-ordering is the number of neighbors of v that are before v 
in the ordering. 

To create an elimination ordering the chompack.maxchardsearch{) function from 
the Chordal Matrix Package (CHOMPACK) El has been used. 



3.4 Minimum Fill-in algorithm 

The minimum fill-in heuristic Q works similarly with minimum degree heuristic, 
but now the vertex v is selected such that the number of edges that is added to make 
a neighborhood of v a clique is as small as possible. 



3.5 Lexicographic breadth-first search algorithm 



Lexicographic breadth-first search algorithm (LEX-BFS) pT] numbers the vertices 
from « to 1 in the order that they are selected. This numbering fixes the positions 
of an elimination scheme. For each vertex v, the label of v will consist of a set 
of numbers listed in decreasing order. The vertices can then be lexicographically 
ordered according to their labels. 
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4 Benchmarking 

4.1 NSDP algorithm implementation 

The NSDP algorithm was implemented by the first author in Python. The ND and 
MD algorithms were implemented in C and C++, respectively. To work with graph 
objects was taken class networkx.Graph from the networkx library ||9l- 



4.2 Test problems 

For benchmarking the DO test problems were generated by using hypergraphs from 
the CSFd hypergraph library [SJ. This collection contains various classes of con- 
straint hypergraphs from industry (DaimlerChrysler, NASA, ISCAS) as well as syn- 
thetically generated ones (e.g. Grid or Cliques). 

The test problems were generated in the following way. The constraints structure of 
a linear DO problem with binary variables was described by hypergraph from the 
library fS) . To build constraint / the next hyperedge of hypergraph was taken, which 
includes a set of variables Xs- for a new building constraint. In the next step, the 
coefficients for appropriate variables of A^^ were generated using a random number 
generator. Then the left part of /-th constraint had view A5.X5., while the right part 
was <tL^5,' where CJ is random number from interval (0, 1). Objective function 
is linear and includes all variables - vertices of hypergraph, where coefficients Cj 
of objective function L/=i CjXj — )■ max where created with help of random number 
generator 

After the test problems were generated, the ordering algorithms MD, ND, MCS, 
MIN-FILL and LEX-BFS were applied for obtaining an elimination ordering. Then 
the problems were solved with the NSDP algorithm by utilizing to the specified 
elimination ordering. 



4.3 Benchmarking ordering analysis 

The following five groups of 33 test problems have been taken: 'dubois', 'bridge', 
'adder', 'pret' and 'NewSystem'. All experimental results were obtained on a ma- 
chine with Intel Core 2 Duo processor 2.66 GHz, 2 GB main memory and operat- 
ing system Linux, version 2.6.35-24-generic. The results can be found in table 1, in 
which n denotes the number of variables, m the number of constraints and the min- 
imal time of problem solving for appropriate heuristics was underlined. We can see 
that for ND algorithm the minimal run-time of the NSDP algorithm was achieved 
times (0 %), for MD 2 times (6,0 %), LEX-BFS 3 times (9,1 %), MCS 9 times (27,3 



■ CSP - Constraint Satisfaction Problem. 
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%) andMIN-FILL 19 times (57,6 %). 
Let us take a look at the benchmarking results in more detail. Fig. [T] describes 



Table 1 Run-time (in seconds) of solving DO problems with help of NSDP algorithm using dif- 
ferent orderings. 



Test 


n 


m 


MD 


ND 


MCS MIN-FILL LEX-BFS 


dubois20 


60 


40 


1,31 


1,43 


1,17 


1,19 


1,20 


dubois21 


63 


42 


1,52 


1,67 


1,35 


1,31 


1,35 


dubois22 


66 


44 


1,37 


1,70 


1,51 


1,51 


1,49 


dubois23 


69 


46 


1,90 


2,02 


1,70 


1,68 


1,74 


dubois24 


72 


48 


2,18 


2,17 


1,89 


1,80 


1,95 


dubois25 


75 


50 


2,72 


2,50 


2,11 


2,14 


2,15 


dubois26 


78 


52 


2,62 


2,82 


2,32 


2,43 


2,39 


dubois27 


81 


54 


2,58 


3,09 


2,60 


2,71 


2,66 


dubois28 


84 


56 


3,55 


3,43 


2,90 


2,84 


2,98 


dubois29 


87 


58 


3,91 


3,84 


3,22 


3,21 


3,28 


dubois30 


90 


60 


4,46 


4,14 


3,50 


3,52 


3,48 


dubois50 


150 100 


17,52 


16,31 


14,34 


14,00 


14,53 


duboislOO 


300 200 


126,32 


111,64 


106,34 


103,23 


106,29 


adder. 15 


106 


75 


6,20 


7,04 


5,25 


5,64 


5,38 


adder_25 


176 125 


27,33 


32,13 


21,47 


24,27 


23,10 


adder_50 


351 250 


326,54 


388,74 


268,39 


276,47 


254,25 


adder_75 


526 375 4876,74 5180,46 3381,28 


3460,32 


3435,76 


bridge. 15 


137 


135 


15,21 


15,41 


13,90 


12,45 


12,86 


bridge_25 


227 225 


63,14 


74,18 


62,30 


55,98 


58,62 


bridge.50 


452 450 


900,20 


983,40 


922,68 


886,91 


1003,89 


bridge_75 


677 675 4832,55 5049,61 4507,38 


3886,34 


5040,91 


pret60.25 


60 


53 


1,89 


1,51 


1,68 


1,32 


1,59 


pret60_40 


60 


53 


1,50 


1,48 


1,69 


1,26 


1,52 


pret60.60 


60 


53 


1,58 


1,51 


1,83 


1,27 


1,64 


pret60_75 


60 


53 


1,44 


1,60 


1,79 


1,31 


1,54 


pretl50.25 


150 133 


21,29 


33,06 


20,51 


16,25 


23,04 


pretl50.40 


150 133 


22,63 


32,83 


21,47 


16,44 


24,76 


pretl50.60 


150 133 


22,09 


31,13 


23,72 


16,35 


23,99 


pretl50.75 


150 133 


21,20 


33,23 


20,19 


18,57 


23,59 


NewSysteml 


142 


85 


19,21 


16,24 


14,05 


17,33 


14,08 


NewSystem2 


345 200 


603,83 


520,82 


376,33 


489,80 


425,35 


NewSystemS 


474 284 1294,86 


1352,75 


1159,75 


1247,81 


1072,58 


NewSystem4 


718 422 7769,10 8322,58 6427,85 


7095,17 


6845,20 



results of the experiment for the groups of test problems 'dubois', 'bridge', and 
'adder'. These results show to us that MCS, MIN-FILL and LEX-BFS heuristics 
behave quite similar and give the best result for a given group of problems. At the 
same time MD and ND show the worst time result. Also for the group 'bridge' we 
can see the gradual decreasing of the time result for the LEX-BFS heuristics and the 
obvious domination of MlN-FlLL. 

Fig. 12] describes experimental results for the groups of test problems 'pret' and 
'NewSystem'. Here we can see the importance of the right choice of heuristics for 



Alexander Sviridenko and Oleg Shcherbina 



Run -time (in percentage) for the group of test? 'dubois' 




# # .# .# # # .# i^ .#■ .# # # -, 

iiT ij?^ ^*^ ^*^ ^ i^ i^ iiT ^*^ i}*^ «*^ «^ i^ 



Run-time (in percentage) fertile group of tests 'adder' 




-MCS 

-MIN-FILL 
-LEX-BFS 



17 
16- 



Run-time [in percentage) for the group oftests 'bridge' 




-MCS 

-MiN-FiLL 
-LEX-BFS 



Fig. 1 Run-time (in seconds) for groups of tests 'dubois', 'adder', 'bridge' 



a certain group of problems. In the case of 'pret', we see obvious domination of 
MIN-FILL algorithm, while MCS and LEX-BFS fall behind. However, the group 
'NewSystem' shows completely opposite results, where MIN-FILL runs third while 
MCS and LEX-BFS take the first two places. 

In the case of 'pret', we see the obvious domination of the MIN-FILL algorithm, 
while MCS and LEX-BFS go back. But the group 'NewSystem' shows the com- 
pletely opposite result, where MIN-FILL goes on the third place while MCS and 
LEX-BFS take the first places. 
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Run-tJme [in percentage) for the group of tests 'pret' 




0' # # # # 
/ / ,*" / y „.«-^ 






-MD 
-ND 
UCS 
-MIM-FILL 
- LEX-BFS 



27 
25 
23 

21 
19 
17 
15 


Run-time [in percentage) for the group of tests 'NewSystem' 




— — ^^^^^^_^ 


-■-ND 
-A-MCS 
i-H-MM-FLL 
-*-LEX^FS 


__;:::;;— =:C^___ * 


..---'^r><? 




"^^^-^-^--'^ 


MewSyalemI MewSystemS MewSyatemS htewSyalem^ 



Fig. 2 Run-time (in seconds) for groups of tests 'pret' and 'NewSystem' 



5 Conclusion 

The goal of this paper to research the role of five variable ordering algorithms and to 
describe the effect that they play on solving time of sparse DO problems with help of 
NSDP algorithm. Our computational experiments demonstrate that, for solving DO 
problems, variable ordering has a significant impact on the run-time for solving the 
problem. Furthermore, different ordering heuristics were observed to be more effec- 
tive for different classes of problems. Overall, the MCS and MIN-FILL heuristics 
have provided the best results for solving DO problems of the problem classes that 
were considered in this paper. It seems promising to continue this line of research 
by studying methods of block elimination with suitable partitioning methods. 
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